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As scientific simulation software becomes more complicated, the 
scientific-software implementor's need for component tests from 
new model developers becomes more crucial. The community's 
ability to follow the basic premise of the Scientific Method 
requires independently repeatable experiments, and model 
Innovators are In the best position to create these test fixtures. 
Scientific software developers also need to quickly judge the 
value of the new model, I.e., its cost-to-benefit ratio in terms 
of gains provided by the new model and implementation risks 
such as cost, time, and quality. 

This paper asks two questions. The first is whether other 
scientific software developers would find published component 
tests useful, and the second is whether model innovators think 
publishing test fixtures is a feasible approach. 

1 Introduction 

This paper is the second installment of the discussion begun 
by AIAA Paper 2004-2627, “CFD: A Castle in the Sand?”, 
which argues that software unit-testing practices are essential 
for advancing computational simulation capabilities as models 
become more complex.^ That paper called for model and 
algorithm innovators to publish succinct test fixtures so that 
subsequent implementors can independently verify they have 
correctly translated the new innovation to source code, i.e., 
so the Scientific Method’s notion of independently- verifiable 
experiments can be used. This installment provides an 
alternative presentation of those ideas in light of copious 
feedback generated by the first paper. 

As growth in computational power facilitates higher-fidelity 
computational simulation techniques, the number and variety 
of building-block components also increases. While this 
increased complexity is forcing a change from the cottage 
industry of one person/one code to team software development 
to address increasing software system size,^ the community 
is not yet routinely publishing independently verifiable tests 
for new models or algorithms to address the code-verification 
complexity. The survey results of table 1 show that only 22% 
of new models published are accompanied by tests suitable for 
independently verifying the new model. 

To sustain our growing numerical simulation capability, 
we need to become competent software developers;^ and 
one measure of software development competence is sound 
software testing practices. For example, before inserting a new 
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Table 1: Survey of new component 
publishing that includes the percent 
of articles introducing new models 
that contain tests. Upticks indi- 
cate articles with component tests, 
downticks indicate articles lacking 
component tests, and dots indicate 
articles that did not appear to intro- 
duce a new model. 


journal 

vol(#) 

articles 

% 

JCP 

192(2) 

iri'iii"i'i'i 

0 


192(1) 


23 


191(2) 


27 

IJNMF 

43 ( 10 - 11 ) 


0 


43(9) 


20 


43(8) 


67 




22 


^ James J. Quirk, “Gomputational Sci- 
ence: Same Old Silence, Same Old 
Mistakes, Something More is Needed”, 
in Adaptive Mesh Refinement - The- 
ory and Applications, edited by Tomaz 
Plewa, et al.. Springer- Verlag, 2004, 

pp. 1 - 26 . 


component into a system, software developers will perform a 
set of component-level tests. Based on feedback from the first 
paper, ^ everyone agrees with the need for component-level 
testing in the computational simulation community but there 
is disagreement about how to implement it. 

While each development group could independently derive 
component-level tests for each model they choose to imple- 
ment, this duplication is unnecessary and would not likely 
catch the special cases that the original innovator would 
likely know intimately. Besides, the Hatton studies of scien- 
tific codes underscores the difficulty in achieving consistent 
implementations: 1 fault per 170 lines.® 

This paper calls for institutionalizing component-level 
testing in the computational simulation community and offers 
one possible route toward implementation. The paper begins 
by exploring the current practice, recalls basic tenets of the 
Scientific Method, proposes a course of action, gives a couple 
brief examples, and finishes with some concluding remarks. 

2 Current Practice 

For the sake of discussion, consider the components of a 
Computational Fluid Dynamics (CFD) code. While developing 
such a code, a team will pull components, such as flux functions, 
boundary conditions, turbulence models, transition models, 
gas chemistry models, data structures, and so on — each from 
a different original publication. For example, consider 24 
components that comprise the FUN3D flow solver® listed in 
table 2. Now, consider the potential interactions between 
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Table 2: Components in the FUN3D flow solver. Data provided by Eric Nielsen of NASA. 


Turbulence model 
Flux reconstruction 
Entropy fix 
Time integration 
Multiprocessing 
Grid sequencing 


Transition model 
Time relaxation 
Transport properties 
Preconditioners 
Domain decomposition 
Grid adaptation 


Boundary conditions 
Gonvergence acceleration 
Data structures 
Flux Jacobians 
Preprocessing 
Grid movement 


Flux limiter 
Flux functions 
Gas chemistry 
Governing equations 
Postprocessing 
Load balancing 


these components as indicated by the lines in figure 1. 
While arguments can be made about whether all components 
necessarily influence all the other components (as drawn), even 
the most ardent detractor has to concede that this system is 
nevertheless a complicated set of interrelated components. 

As the number of components increases, the potential inter- 
actions grow as n^/2, where n is the number of components. 
The task of finding an error in a system of interrelated com- 
ponents is daunting, but this task becomes untenable if the 
components have not been independently verified. Rational 
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Figure 1: Potential component interactions in the FUN3D flow solver. 


verification of this complicated system must proceed in two 
steps: (1) verification of components and (2) verification of 
their interactions. 

The current computational verification and validation 
literature recommends verification on the system level by 
using the Method of Manufactured Solutions (MMS).'^ While 
this is a necessary step in every code- verification process, it has 
not yet been widely practiced due to implementation overhead® 
and because if this system-level test fails, the debugging task 
could be in any of n components in addition to the roughly 
n?/2 component interactions. Therefore, before attempting 
MMS on a system of components, each component should be 
independently verified. 

Consider for example, the dilemma created by the de- 
but publication of the popular Spalart-Allmaras turbulence 
model.® The document contains a mathematical description 
of the model and then shows comparisons with experimental 
boundary layer profiles that require a complete CFD code 
system. This scenario is sketched in figure 2, in which New 
Component is the mathematical description of the new turbu- 
lence model and the author’s code are indicated by Component 
Code A and System Code A. The boundary layer profile output 
appears at the bottom. 

The dilemma is that no isolated tests of the turbulence 
model itself, either mathematical or numerical, are presented. 
So, when another CFD code development team (path B) 
elects to install this new model in their system, a comparison 
with boundary layer profiles does not assure the model was 
implemented in the same way as the original because the other 
code components are completely different. The specific effects 
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Figure 2: Current method of translating 
the “paper” model to numerical results. 
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of the turbulence model become lost in the large computational 
simulation infrastructure, and there is no credible means to 
determine that both codes are using precisely the same model. 
As a consequence of this implementation risk and the lack of 
test data, the implementor is unable to quickly determine the 
value of the new model. 


3 The Scientific Method 

In a computational context, component-based verification 
testing is the engine behind the Scientific Method that Roger 
Bacon first described in the thirteenth century: a repeating 
cycle of observation, hypothesis, experimentation, and the need 
for independent verifieation}^ 

Popularized by Francis Bacon and Galileo Galilei, the 
Scientific Method has since become a means of differentiating 
science from pseudoscience. The Scientific Method is fueled 
by the idea that hypotheses and models must be presented to 
the community along with the description of experiments that 
support the hypothesis. The experiments that accompany 
a hypothesis must be documented to the extent that others 
can repeat the experiment — Roger Bacon’s independent 
verification. 

This last notion, that others should be able to repeat an 


Roger Bacon, Opii: Majus, Minus, 
and Tertium, c.1267. 



Figure 2: Current method (repeated for convenience). 
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experiment is currently not widely practiced by computer 
simulation software community. In part, this is due to the 
large body of software required by a modern simulation 
capability. While electronic documentation methods such as 
Quirk’s Amrita system^^ can go a long way toward solving this 
issue, the fact remains that our experiments must be small 
enough and isolated enough to be independently repeatable 
and widely applicable. Ultimately, an implementor should be 
able to come to the same conclusion as another implementor 
based solely on the numerical evidence. 

4 Proposed Practice 

How can the computational simulation community leverage 
the Scientific Method? — by having innovators publish a set 
of tests when they present a new model or algorithm so 
implementors can gage the innovation’s value and reliably 
make the transition from the mathematics to the numerics. 
This notion is depicted by the pages labeled Component 
Verification in figure 3, where model innovators publish 
component test fixtures so that developer B can verify the 
numerical implementation of the mathematical model or 
algorithm in isolation before inserting it into her simulation 
system. The tests, or numerical experiments, should consist 


11 www.amrita-cfd.org 
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Figure 3: Proposed method of translating the “paper” model to numerical results. 
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of simple input/output combinations that document the 
behavior of the model. In particular, boundary cases and 
any other special cases should be documented. For example, 
the temperature domain of Sutherland’s viscosity law or the 
nonrealizable initial states for a linearized Riemann solver flux 
function. 

Wherever possible, tests should be written at the mathe- 
matical level, but some actual discrete values should also 
be presented. The latter is particularly advantageous if the 
experiments are designed to expose boundary areas that 
are sensitive to divided differences, nonlinear limiters, or 
truncation error. (Examples are given in section 5.) 

All subsequent developers that implement the model and 
publish their results would be required to document which of 
the original verification experiments they conducted. Over 
time, the popular techniques could have a suite of tests 
formally sanctioned by a governing body such as the AIAA 
so that any implementation would have to pass the standard 
tests to be considered verified. Otherwise, the community is 
left to suffer the fate of unquantified uncertainties as described 
by Youden’s two seminal works. 

5 Examples 

The last paper contains examples for the CIR Scheme,^® and 
the Van Albada limiter function. In this paper, two simple 
components are discussed: the Sutherland viscosity law and 
the modified Newtonian law. 

Table 3 shows a succinct component test fixture for Suther- 
land’s viscosity law, which gives the viscosity of air as a 
function of temperature. The mathematical form is presented 
along with boundary points and a value from the middle of 
the domain. While this example is trivial, it demonstrates the 
very localized level at which components should be considered. 

For example, the flux function example presented in the 
previous paper was attacked on the grounds that it would 
be impossible to cover all the discretization settings in which 
it would be applied, e.g., finite volume, finite difference, or 
finite element. These considerations are an indication that the 
component is being defined at too high a level. 

Another component examined is the Modified Newtonian 
law, which gives pressure coefficient as a function of surface 
inclination according to, 

Cp = Clpmax sin^ ^ (1) 

where 0 is the surface inclination angle in degrees, i.e., the 
angle between the incoming flow and the surface normal 
vector. The stagnation pressure coefficient is governed by 
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The one-dimensional version of Roe’s 
Flux Difference Splitting flux function. 


Table 3: Sutherland’s viscosity law 
component test fixture. 


input 

output 

T(K) 

(kg/s-m) 

200 < T < 3000 

rrn 1 . 4 

K* T 
T+110.4 

199 

error 

200 

1.329 xlO^ 

2000 

6.179 xlO^ 

3000 

7.702 xlO^ 

3001 

error 


* where K = 1.458 xlO 
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shock relations 




47M^- 2(7-1) 


7/(7-i) r 


1 - 7 + 2-/M^ 

7 + 1 


- 1 


Table 4: 
fixture. 


where Mqo is the freestream Mach number and 7 is the ratio 
of specific heats. 

A sample component test fixture for this law is shown 
in table 4. Again, it begins by defining the valid input 
domains with pure math. Next, certain limiting cases 
are provided along with a sampling of interior points. 

Finally, boundary cases are shown and suggested error 
messages are given. 

Other examples of component-based testing are 
available for an advection-diffusion solver that was 
written during an exploration of using test-driven 
development for scientific simulation codes. 


Modified Newtonian Law component test 


6 Concluding Remarks 

To sustain our growing numerical simulation capabili- 
ties, we need to become competent software developers 
by increasing our use of component testing practices. 

The implementation path offered here is to have model 
innovators publish simple, component-level verification 
test fixtures so that implementors can verify their 
implementation according to the basic premise of the 
Scientific Method — independently-verifiable experi- 
ments. 

Based on feedback from the first paper in this series, most 
readers agree that component-level testing should be standard 
practice in the computational simulation software development 
community. However, two questions remain: 

Do scientific software developers want published 
component tests? 

Is the proposed solution palatable by model innovators? 



input 


output 

e (deg) 

Moo 

7 

Cp 

0 < 0 < 90 

5 < Moo 

1 < 7 <3 

Eq. 1 

0 

100 

1 

2.000 

45 

100 

1 

0.500 

0 

100 

1.4 

1.839 

0 

5 

1.4 

1.809 

0 

4.9 

V 

error 

91 

V 

V 

error 

-91 

V 

V 

error 


Where V indicates "for all valid values” 
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If the answer to either is “no,” then how should we proceed? 

Unless feedback on this paper dictates otherwise, the next 
installment of this series will present a more detailed example 
by using Test-Driven Development,^® a promising extension of 
agile programming methodologies. 


See for example, 

c2 .com / cgi / wiki?TestDrivenDevelopment , 
last accessed June 1st, 2005. 


7 OF 8 


AIAA Paper 2005-4873 


A cknowledgments 

All the folks mentioned in the first paper^® plus all the folks 
that responded directly during the Portland conference plus 
all the independently received comments/suggestions plus the 
three anonymous journal reviewers for AIAA’s Journal of 
Aerospace Computing, Information, and Communication, and 
David Coppit, Professor of Computer Science at The College 
of William and Mary. 

About the Authors 

Bil Kleb, a lifetime senior AIAA member, has 
worked in the area of computational aerothermo- 
dynamics at NASA’s Langley Research Center for 
the past 15 years. Since 1999, Bil has been active in the 
agile software development community^*^ and has given several 
invited lectures on team software development for scientific 
software. For the past two years, Bil has been a steward of the 
FUN3D software development team^^ and has been serving 
on various agile software development conference committees 
since 2002. 

For those that measure by certificates and degrees, Bil has a 
B.S. and M.S. in Aeronautical and Astronautical Engineering 
from Purdue University, an M.B.A. from The College of 
William and Mary, a Ph.D. of Aerospace Engineering from the 
University of Michigan, and a commercial pilot certificate with 
instrument rating. 

Email: Bil.Kleb@NASA.Gov 

Bill Wood has worked in the field of CED in the 
Aerothermodynamics Branch at NASA’s Langley 
Research Center since 1991, earning a Ph.D. in 
Aerospace Engineering from Virginia Tech in 2001. He served 
on the program committee for the software development 
conference XP/Agile Universe from 2002 to 2004 and is now 
serving on the Agile 2005 conference committee. 

Email: Bill.Wood@NASA.Gov 


a 



Colophon 

This document was typeset in Computer Modern font with 
the DT[^ typesetting system using the handout option of 
the AIAA package, version 3.8. The handout class option 
used here strives toward the layout style espoused by the 
visual information design expert Edward Tufte.^^ Also 
employed were the color, rcsinfo, bibentry, varioref, 
wrapfig, threeparttable, booktabs, wrapfig, hyperref, 
and nohyperref packages. 


8 OF 8 


19 Bil Kleb and Bill Wood, CFD: A 
Castle in the Sand?, AIAA Paper 2004- 
2627, 2004. 


^^For agile software development’s 
succinct, but surprisingly powerful 
manifesto, see agilemanifesto.org. 

fun3d.larc.nasa.gov 


www.ctan.org 

Edward R. Tufte, The Visual Display 
of Quantitative Information, Graphics 
Press, 1983; Edward R. Tufte, Envi- 
sioning Information, Graphics Press, 
1990; and Edward R. Tufte, Visual Ex- 
planations: Images and Quantities, Ev- 
idence and Narrative, Graphics Press, 
1997. 


AIAA Paper 2005-4873 


